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Evaluation models in5)ly or assume theories of 
organization, behavior, and decision-making, seldom does an 
evaluation model specify these assumptions. As a result, program 
evaluators often choose mechanistic models and their resultant 
information is either inadequate or inappropriate for most cf the 
client's purposes. The Ontological Evaluation Model proposed in this 
paper is based on a developmental assumption, namely, that humans in 
purposeful groups, organizations, reenact a cycle of development very 
much akin to the ontology of a single individual. An educational 
program is a special case of human organizational development. 
Program evaluat-.on from this contextual premise, therefore, takes 
organizational development as its structural analog. Reference is 
made in the presentation to applications in which the model has been 
used and to which its use seems most efficient. (Author) 



ED 078 039 

AUTHOR 
TITLE 

PUB DATE 
NOTE 

EDRS PFICE 
DESCRIPTORS 

IDENTIFIERS 
ABSTRACT 



FILMED FROM BEST AVAILABLE COPY 



U$ OE^ARTMENTOFHEAtfH. 
EbUCATlONftwELFAllE 
NATIONAL INSTITUTE OF 
EOUCATlOM 
THIS 00CUV6NT MAS BEEN REPRQ 
OUCEO EXACTLY AS RECEIVEO FRQv 
TME PERSON OR ORC«.NiZATiON ORiCiN 
ATlNClT POlNTSOP VIEW OR OPINIONS 
iTATEO 00 NOT NECESSARILY REPRF 

Sentopficial national institute of 

COUCATiON POStTiON OR POLtCV 



AN ONTOLOGICAL MODEL OF EVALUATION: 
A DYNAMIC MODEL FOR AIDING ORGANIZATIONAL DEVELOPMENT 



John B. Peper, Consultant 
Texas Educational Renewal Center 
6504 Tracor Lane 
Austin, Texas 78721 



Associate Professor 
University of Virginia 
Evaluation Research Center 
Ciu*ry Memorial School of Education 



A Presentation Prepared for the American Educational 
Research Association Annual Meeting in New Orleans, 
Louisiana, February 26 - March 1, 1973 
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Organizations, the way they develop and the impact that they have on 
their participants will be the subject for evaluation efforts in the next 
several years, for organizational development will replace curriculum 
development as the vehicle of change in educ ation. 

This represents a departure from the past decade in which educational ~ 
change was focused on improvements in the school curriculum (Gagne, 1967). 
Almost every subject area was exposed to ne^^ efforts aimed at bringing 
about reforms in public education. Mathematics, science, English and 
social studies all received huge sums of monej- to provide curriculum 
improvements with the ultimate objective directed toward making students 
more capable of responding to a rapidly changir.g world. 

Evaluation was used during the sixties to improve course offerings. 
Chronbach (1964) outlined a methodology for using evaluation in course 
improvement. He suggested a combination of (1) process studies, (2) pro- 
ficiency measures, (3) attitude measures, and (4) follow-up studies. Stating 
that "evaluation is a fundamental part of curriculim development, not an 
appendage, " he advocated a functional view of evaluation, "its job is 
to collect facts the course developer can use and will use to do a better 
job, and facts from which a deeper understanding of the educational 
process will emerge. " 

In the middle and late sixties a host of writers began to develop both 
theoretical and operational models for curriculum development and for 
program development. In most cases, program was seen as an organi- 
zational strategy for installing or improving a specific course area. 



Stufflebeam (1967) suggested the CIPP model. Its acronym indicates four 
types of evaluation; namely, context, input, process and products Stake 
(1967) enunciated a need for "standard ways of translating aims and needs 
into practice. " Tyler (1967) described the course of change during the 
sixties in evaluation f-rom one of using evaluation instruments to screen 
talent to one of using evaluation to enable the learner to make appropriate 
responses to situations outside his controL Interest and attention was also 
devoted to developing new classifications for the evaluation strategies 
which could be used for different purposes* Scriven (1967) offered the 
concepts "formative, " "summative, " "payoff, " "intrinsic, " and "mediated, " 
evaluation. More recently, he has added "goal-free" to the list. Generally 
speaking, the notion running throughout the development of each concept 
was that the technique would be used to improve a curriculum or subject 
oriented program* 

Other models were developed with a curricular emphasis. Provus (1969) 
offered the discrepancy model. Use of the discrepancy model permits a 
program director to define standards for the program and to compare 
those standards with existing practice. In a later version of the model, 
Provus (1971) states that program standards and, hence, evaluation practice 
will vary with level of organizational development. He classifies developmental 
phases of evaluation as (1) design, (2) installation, (3) process, (4) product, 
and (5) cost. Each stage corresponds with a level of program development. 
Bloom and others (1971) published a handbook of formative and summative 
evaluation techniques to be used for evaluation of student learning. 



Considerable literature has been written recently which indicates 

that curriculum oriented change is insufficient in itself to bring about 

needed improvements in education. Blumberg (19Y2) stated the problem 

succinctly as follows: 

The last fifteen years or so have witnessed an 
almost bewildering array of change efforts in 
schools. These have focused on curriculum 
teaching methods, alternative patterns of 
staffing, preschool education, the introduction 
of media technology and management information 
systems, to mention a few. But, in a curious 
way, we often feel there has been a great deal 
of change — but no change ... The point 
seems to be that the recent as well as the '.ong 
term history of educational change has been a 
first-aid affair that has left the basic fibre 
oi schools unchanged. 

a • • Put another way, ... There is a strong 
suggestion that most of it has been wasted 
because the target of it all has been peripheral 
to where the basis of long-lasting and produc- 
tive change may be found — in the nature of 
the human organization or system that is the 
school. (Blumberg, 9) 

One of the most popular analyses of social trends in America has 

proved to be Toffler's Future Shock. Toffler attributes much of our new 

interest in organizations to the accelerated pace of life brought about by 

technological change and by population growth. The trend is labeled as 

"the new ad - hocracy. " 

We are . . . witnessing the arrival of a new 
oi ganizational system that will increasingly 
ch lUenge and ultimately supplant bureaucracy. 
This is the organization of the future. I 
call it "ad - hocracy ''o ... Man will find 
himself liberated, a stranger in a new free- 
form world of kinetic organizations . • • His 
position will be constantly changing, fluid 
and varied. And his organizational ties, 
like his ties with things, places and people. 



will turn over at a frenetic and ever 
accelerating rate. (Toffler, 125) 

Toffler further delineates how he thinks this emphasis on organizational 

change will impact schools of the futureo 

What passes for education today, even in our 
"best" schools and colleges, is a hopeless 
anachronism. . . What has been lacking is a 
consistent direction and a logical starting 
point. . . The starting point: the future. 
Such a movement will have to pursue three 
objectives — to transform the organizational 
structure of the educational system, to 
revolutionalize its curriculum, and to encourage 
a more future - focused orientation. It must 
begin by asking root questions about the status 
quo. 

Attempts have been made in the past to use evaluation as a basic 
component of organizational change, but those efforts inevitably returned 
to programmatic or curriculum emphasis. Brickell (1961) proposed a 
system of organizing New York State Department of Education for change, 
including the use of evaluation as one of the three major strategies for 
introducing change. They were designing programs, evaluating programs 
and disseminating programs. Program was used by Brickell to refer to 
a curriculum offering. Cook (1966) adapted the United States Air Force 
PERT manual to the educational environment, but again emphasis in 
Cook's technique was educational program. 

One present need in evaluation is to develop an evaluation model which 
makes the organization its focus and not a single program within an 
organizationo The ontological evaluation model is proposed for the 
specific purpose of providing a conceptual framework through which 



evaluation can occur in and about developing organizations. Ontological 

development is stressed as a point of view which encourages a purposeful 

system of analyzing the dynamic and ever changing state of an organization. 

Too often in the past evaluation has proceeded as if the object of study 

were in a stable state. Almost all organizations are in continuous change. 

Grobman (1968) acknowledged that program focused evaluation has 

achieved little identifiable impact. 

In the past decade we have seen educational 
innovation on a larger scale than ever before. 
Much of the innovation effort has been 
devoted to curriculum development. While 
evaluation has played a part in many of the 
Course Content Improvement Projects, we 
have a sketchy record, at best, of what 
it contributed to the projects it was 
commissioned to serve. (Grobman, p. vii. ) 

In a more recent article. Walker (1972) made a similar acknowledgement* 

Walker further stated that evaluation is unlikely to have any impact unless 

it is tied directly to an identifiable series of decisions. It is unlikely 

that program evaluation will have a serious impact on organizational 

decision-making until evaluation is able to embed itself in the decision 

structure of the organization. To accomplish this a model is needed which 

will assist evaluators comprehend their roles and functions with reference 

to a specific organizational client. 



The Ontological View of an Organization 

The theory of scie^.ce has developed two positions with quite 
different implications for research, scientific development and evaluation. 
Those positions have been explicated in detail by Ayres (1969) in his 
book Technological Forecasting and Long-Range Planning. Basically, 
the ontological view holds that an organization or any organism for that 
matter has the ultimate capacity for change within and that permanent 
development can best be explained by focusing on the nature of the organi- 
zation rather than in terms of external forces. Conversely, the teleologi cal 
viewpoint attributes change in an organization to external forces and stimuli. 

Currently there is a serious debate among researchers and other 
scientists over which approach holds most value. The fundamental dif- 
ference hinges on the way that an investigation is perceived. 

Scientists who proceed from the teleological position maintain that goals 
should be predetermined and that invention and discovery are to be tolerated 
only as they fit into the predefined or "engineered'* plan as approved at the 
outset. From this position it is assumed that development can only proceed 
to each succeeding stage in a linear fashion. As one phase is completed 
and proved causal in a chain of events then the next stage may be under- 
taken. Program development would proceed only on the basis of past 
research or evaluation and would be the result of a deductively derived 
series of causally proven steps. 

Scientists who value discovery and systems which permit an account- 
ing of unanticipated events operate from the ontological perspective, 

^ f 
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This perspective permits investigation and development to occur in a 
systematic pattern of both parallel and serial events. Evaluation or re- 
search may be integral to and concurrent with development from this 
perspective. It is the evolutionary development of an organization that 
is valued and assisted by ontological evaluation. Stake (1972) provided 
an excellent insight into this position through his proposal for ''responsive 
evaluation*' which he compares with "preordinate evaluation". An 
evaluation is preordinate if it emphasizes ''(1) a statement of goals, 
(2) use of objective tests, (3) standards held by program personnel, and 
(4) research-tjrpe reports. '' An evaluation is responsive if it "orients 
more directly to program activities than to program intents, if it responds 
to audience requirements for information, and if the different value - 
perspectives present are referred to in reporting the success of the 
program." 

In a sense Scriven (1972) advocates the ontological point of view when 
he describes goal free evaluation, "it seemed to me, in short, that 
consideration and evaluation of goals was an unnecessary but also a 
possibly contaminating step. I began to work on an alternative approach — 
simply, the evaluation of actual effects against (tjrpically) a profile of 
demonstrated needs in this region of education. (This is close to what 
Consumers' Union actually does.) I call this Goal-Free Evaluation (GFE). " 

A more definitive understanding of the ontological position is possible 
through a brief review of its philosophical and theoretical development. 
In Greek philosophy ontology refers to the philosophy of existence. 
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Plato believed that existence was one of the things that the mind apprehends 
by itselfo Anselm espoused the ontological argument for the existence of 
God as the greatest of all possible thoughts. Leibniz used the ontological 
position as did HegeU Theologians such as Aquinas rejected the position. 
In the theory of science Bacon used the ontological position to support his 
belief that an inductive rather than a deductive approach was more valua- 
ble to scientific investigation. 

In a similar fashion geologists approach their research from the 
positions of ^'analysis" and "synthesis Those who hold the "analysis'' 
view explain geological phenomena from a preordinate or teleological 
view point. Those who proceed with the "synthesis" point of view explain 
phenomena through processes of recreation and discovery. In the evalua- 
tion model proposed below, the organization is considered to be the object 
of study, and the approach or perspective offered is that one can only 
create knowledge of use to the orgsuiization by becoming part of the 
developmental process. 

Typically there is a creative search for intrinsic or endogenous 
variables rather than an imposed design. The kind of study that would 
emerge could easily resemble systems analysis versus an imposed 
agricultural experimental design. In all probability ontological (^valuation 
will chart what an organization is becoming better than it will describe 
what it is. 
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The Ontological Model for Organizational Evaluation 

Organizational Characteristics 

The figure on the following page contains a conceptual drawing of a 
three dimensional model. In order to stress the organization as the 
object of interest an analog of human development has been purposefully 
chosen. Just as the ontogeny of a single individual has identifiable but 
interactive phases and forms, so does the ontology of an organization. 

Morphology or Structure . Morphology or structure has been chosen 
to represent a requirement of the evaluator to become awar.. of components, 
shape, boundaries and texture. Ayres (1969) uses a technique called 
morphological analysis to chart and graph the probable future of an 
organization through analysis of its structure. 

Systems , The systems of an organization interact with structure, 
thrust and kinship linkages to provide an assessment of the internal 
workings of an organization. Their analogues to human systems are 
direct. Information systems are similar in operation among organizations 
to the cognitive processes of a single individual. Furthermore, it is 
difficult to understand an organization without being able to conceptualize 
interactions among the cognitive systems and other systems within the 
organization. Sentiment systems must be assessed as they generate and 
affect values and drives of the organization^ and communications systems 
relate to the direction, control, dissemination and persuasive activities 
of an organization. Resource generation and consumption are generalizable 
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systems but are unique to almost every organization^ Facilities systems 
are separate from morphology in that an organization has continuous 
developmental struggles with logistics, occupancy, assignment, utilization 
and location of facilities. 

Kinesthesia or Thrust. Development of organizational goals and 
objectives can be viewed as a dynamic process. In this sense the word 
thrust is more appropriate. Oae can note direction such as reading or 
mathematics or public health, but consequence objectives are almost 
always useless in changing the course of events for those on whom the 
measures were taken. What is required is a system of projecting the future 
since it is only in the present and in the future that one has any meaningful 
control of events. One then must be conscious of determining direction, 
rate, source of force fields and possible endurance characteristics of 
the organization given present and projected thrusts. 

Kinship Linkages 

Almost every evaluator finds it necessary to understand the types 
of relationships that exist within and between organizations in order to 
operate effectively. The kinship linkages are classified in two ways on the 
following page. Evaluators must learn where in the organization to get 
information and how that information is generated. To do this they need 
to know the possible and actual channels of communication. Formal 
charts are sometimes helpful, but m emergent organizations they are not 
likely to provide sufficient information. One of the first tasks of the 
consultant is to map kinship ties even if he does this subconsciously. 
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The between and within group communication channels have both 
amplification and impedence characteristics as well as problems created 
by extraneous or "noise'' signals. In addition, kinship linkages may be 
characterized by the relationship tie and its vertical or horizontal 
directions of message flow. Power and authority relationships tend to 
foster vertical communications and only among peers are the communications 
likely to be horizontal. One can see the importance of this analysis when 
a low power figure relates to a high power figure horizontally. There 
are usually other explanations that bear further analysis. 

Phases of Organizational Development 

As an emerging profession, evaluation has a relatively brief 
history. It was not uncommon for evaluators to be expected in the 
early days to do the impossible. They would be asked to 'deliver an 
evaluation report of a project long before the project was organized or 
before it had an identifiable form or thrust. It became necessary, there- 
fore, to be able to identify organizational stages of development con- 
currently with developing other assessments. Figure 3 provides a 
conceptualization of five stages of organizational developn ento A pre - 
emergent organization may be a conceptual plan such as a project awaiting 
funding or a school staff during building construction. The emergent 
organization is one which is just beginning to develop but it is without 
an identifiable form or thrust. The familial dependent organization is 
dependent on a parent organization for resources, but has a distinct 
form and thrust and can be differentiated from other organizations in the 
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same general family. The fully independent organization is capable of 
regeneration and is dependent on the environment only for resourceSo Form 
and thrust are both carefully delineated and identifiable • 

Evaluation Functions 

Evaluation functions are sepsirated in the ontological model from the 
organizational characteristics and phases of development only as a way 
of conceptualizing them. Evaluation functions may, in fact, be imbedded 
within the normal systems of the organization. One of the paramount 
difficulties that the evaluator confronts within an organization is that of 
clarifying the differences between the evaluative functions and who has 
responsibility for performing these functions. For example, it is often 
necessary to use existing data for evaluative purposes, MisunderstandLigs 
can occur if the person who originally -collected the data for other purposes 
feels that his jurisdictional or territorial rights have been invaded,, 

Consultation , The above and similar problems make client consul- 
tation the primary function of the evaluator. Without the capacity to 
negotiate design and data problems as well as the ability to appreciate 
the client's position, the evaluator will find all of the other functions mean- 
ingless and in all likelihood, unmanageable. The Evaluation Research 
Center at the University of Virginia approaches every student training 
activity with the client-evaluator role as paramount. That kind of clinical 
approach forces the staff to learn more than the students in many cases, 
but it also emphasizes for students the necessity of learning to deal 
effectively with clients, 
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Planning. Almost every organizational administrator requires plan- 
ning assistance. Since e valuators typically have analytical minds and 
quantification skills, it is also assumed that they will be able to assist 
management in planning. They may not, however, be good planners in 
a particular organizational setting. It is more important that the plan- 
ning function occur than that the evaluator do the planning. Planning at 
its best in a developing organization is a tool for identifying organizational 
thrusts and resources, and is the scheduling of the utilization of resources 
to proceed toward identified thrusts while using new information to plan 
other new thrusts and resource utilization. The gerund is an appropriate 
syntactical description of this function since planning is never complete, 
but always dynamic and regenerative. 

Observation. How observations take place and what is observed are 
among the most important functions of evaluation. From the ontological 
perspective the evaluator cannot have preconceived plans of observation^ 
He must develop the observation system to fit the requirements of emerging 
variables which he discovers in the process of coming to know an 
organization. Reading sources may be far less important than whether 
children choose to read a newspaper when given the opportunity^ In 
creating an observation system for teacher education, it is more impor- 
tant to have that system improve teaching than it is to have performance 
indices on a graph or chart. Who does the observing and the observational 
media to be used are also developmental questions which emerge from the 
organization from this perspective. 



Data Collection ^ One could easily postulate the data collection position 
of this model. Data are generated in most organizations in large quantities. 
If one knows the data generation stations and can use existing data, 
teachers who give examinations as part of their normal course of instruction 
would welcome use of those tests in evaluative processes^ Administrators 
often wonder why artificial means are developed to collect data they 
routinely have in their files. There are times, however, when data 
collection for evaluation must occur outside or in addition to the other 
functions of the organization. When this occurs careful plans must be 
developed to provide necessary resources for this function. Every 
evaluator has a portfolio of war stories to tell in this regard. 

Data Edit or Reduction. Data may come in the form of movie tapes, 
television recordings, recordings, narrative reports, formal tests and in 
a host of other forms. The editing or reduction can be one of the most 
difficult to manage. It consists of reducing observations to a manageable 
and a compatible format for later analysis. Scoring of tests, tape editing 
and content analysis coding of narrative records are all required skills 
in this function. 

Data Analysis. Experimental design and other teleological scientific 
approaches to analysis of data demand that the analysis system be 
announced and planiied in detail prior to the study. Qntological evaluation 
maintains that it is not only appropriate, but more scientific to plan the 
analysis of emergent data in the social sciences during or following the 
collection. Some evaluator s apologetically admit that they use descriptive 



statistics and correlational formulas for many of the?x numerical data. 
The myth that experimental design is more scientific than emergent 
design forms in the social sciences has to be placed along side alchemy 
as a long outworn fable. It is more scientific to use emergent analysis 
techniques to capture the dynamics of change than it is to make false 
assumption that people change no more than plots of ground with two kinds 
of fertilizer applied. Experimental design is a static engineering 
technique with limited purposes and no more. One of the most damaging 
pieces of thinking that has occurred is the notion that the next best thing 
to an experimental design is a quasi-experimental design. Designs for 
evaluation and research in the social sciences would do well to disregard 
experimental design in the statistical sense entirely. Far more time 
should be devoted to conceptualizing the use of descriptive data in creative 
formats to generate powerful and useful inferences. Data analysis is far 
less important than the validity of the data going into the analysis. Path 
analysis and intuitive grouping lend themselves to hypothesis generation. 
Generation of plausible hypotheses about the future based on current 
decisions is in the end the most worthwhile of evaluation functions. 

Data Synthesis. Displays of data require careful thought because it is 
the s;)'Tithesis function used to postulate the major forecasts and recom- 
mendations for decision makers. Data cost in an aggregate form which says 
that a million dcllar effort and two years of work by twenty people was 
unsuccessful generally ends up helping no one. What is required is to 
be able to forecast kinds of change needed to carry on the general thrusts 
in the future. Society does t quit trying to find cures for cancer because 



previous efforts have failed. What everyone needs is information which 
will assist in the next round of studies. Evaluation properly done from 
the ontological position could make such a statement. 

Report Preparation. Report preparation includes a number of choices 
about format, audience, timing, impact, purpose and class of decision 
influenced by the report. Each consideration carries a series of logical 
activities with it. Preparation of a report should take into account the 
possible unanticipated effects of the particular presentation. 

Report presentation. . Mailing a written report to a decision-maker 
is a common practice, and it is the least likely method of influencing a 
decision outside sending no report at all. Reports should be presented 
in face to face meetings where discussion and interaction occur both about 
form and substance. If the presentation is timed to fit into the decision 
flow of an organization, a direct impact can come into the decision network. 



The Evaluator 

All of the foregoing material has been written without a major 
discussion of the evaluator and his role. A deliberate approach has been 
undertaken to explicate the functions of evaluation prior to a discussion 
of the evaluator' s role. In the ontological model of evaluation, the 
evaluator could operate both internally and externally with reference to 
the organization he is attempting to assist. He must be conscious of 
the model's values, however. That means a commitment to use his role 
to assist the organization in its development. His activities will be 
characterized by a careful search for intrinsic or endogenous variables. 
He will intentionally try to design his observation system and reporting 
system to place him in a position to make recommendations about the 
future rather than assume the position of trying to make pass or fail 
judgments about past performance of the organization. 

Client centered consultation about the organization will typify the 
relationship. He will frequently be asked to report on emerging but 
unanticipated phenomena, and evaluators using the ontological perspective 
will want to elicit variables rather than impose them. It is also possible 
that the evaluator from this position would prefer induction ^( reduction. 

VTlien evaluator role is discussed, it is common to confront the ob- 
jectivity question* When speaking to laymen, they generally imply or 
state directly that they think that the evaluator /ould be more objective 
if located outside an organization than if he operated from within. No 



7 I 



doubt the external evaluator can remain more dispassionate the further 
he is removed from the o.oganization, but his observations will not be 
more accurate. 

Two analogies can be drawn. First, it is impossible to observe with 
a microscope and with a telescope. Neither observation device assures 
objectivity nor a lack of it. Second, a photographer can use the same 
camera to photograph a parlor from inside a house that he uses to 
photograph the roof of that same house. Neither photograph exhausts 
the concept 'house', but both can be recognized as pictures of a house. 
Moreover, there is a noticeable interactive effect in social evaluation 
between evaluator and the organization being served. Location of the 
evaluator does not necessarily preclude nor guarantee the presence of 
objectivity. Proximity has a potential for enhancir^ the interaction 
between evaluator and client. The position which would lend itself to the 
possibility of generating the largest quantity of information about an 
organization would appoar to be one of close proximity. 



Summary 

A three dimensional conceptual model has been proposed for con- 
ducting evaluation of organizations versus programs or curricula* The 
ontological position is advocated in order to promote the dynamic search 
for intrinsic or endogenous variables. The e valuator uses the functions 
of evaluation to assist a devr^loping organization in its ontology or state 
of becoming. Finally, it is proposed that the emphasis on the developing 
organization is more important than the position of the evaluator inside 
or outs?de an organization. But acknowledgement is also made of the 
fact that position does influence perspective and client- evaluator interaction. 
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